Method and system for enhanced image sensor timing

ABSTRACT

A stereoscopic image capture device includes a first image sensor, a second image sensor, a first frame timer, and a second frame timer. The first and second frame timers are different frame timers. The first image sensor includes a first plurality of rows of pixels. The second image sensor includes a second plurality of rows of pixels. The first and second image sensors can be separate devices or different areas of a sensor region in an integrated circuit. The first frame timer is coupled to the first image sensor to provide image capture timing signals to the first image sensor. The second frame timer coupled to the second image sensor to provide image capture timing signals to the second image sensor.

RELATED APPLICATIONS

The present application is a U.S. National Stage Application under 35U.S.C. § 371 of International Application No. PCT/US2019/051593, filedon Sep. 17, 2019, which claims priority to U.S. Provisional PatentApplication No. 62/732,718, filed on Sep. 18, 2018, and entitled “METHODAND SYSTEM FOR ENHANCED IMAGE SENSOR TIMING,” the contents of which arehereby incorporated by reference in their entirety.

BACKGROUND Field of Invention

Aspects of this invention are related to medical device imaging, andmore particularly to combinations of normal imaging and advancedimaging.

Related Art

Image capture devices are used in minimally invasive surgery. Variousimaging modalities—visible scenes, fluorescence scenes, infrared scenes,hyperspectral scenes—are implemented using image capture devices.However, each imaging modality utilizes one or more parameters, e.g.,exposure time, that are different from the corresponding one or moreparameters in the other imaging modalities. This makes using an imagesensor configured for one imaging modality difficult to use for adifferent imaging modality.

The problem of using a single image sensor for different imagingmodalities is further complicated when a stereoscopic image capturesystem is used and the image sensors have been optimized to capturevisible color scenes. As is known, an image sensor includes pixels thatcapture and integrate light over time. To maximize the area of the chipavailable for pixels, other circuitry on the image sensor is kept to aminimum.

For example in a stereoscopic complementary metal-oxide-semiconductor(CMOS) sensor integrated circuit, the sensor region is divided into twoareas, a first area includes pixels that capture a left scene and asecond area includes pixels that capture a right scene. Both areas ofthe sensor region have the pixels arranged in rows and columns. There isa reset line and a row select line associated with each row of thesensor region, and a read line associated with each pixel of each row ofthe sensor region. To minimize the logic required on the integratedcircuit, a common frame timer logic circuit is used to drive the resetand row select lines of both sensor regions.

FIG. 1 is a timing diagram for a CMOS sensor integrated circuit thatutilizes a rolling shutter to capture a frame of pixel data. The timingdiagram is the same for both channels of a stereoscopic image capturedevice. In FIG. 1 , an Nth frame 101 is captured, followed by an N+1frame 102, and by an N+2 frame 103. N+1 frame 102 is sometimes referredto as frame 102.

In this example, the capture of line zero of N+1 frame 102 isconsidered. (A line of pixels and a row of pixels are the same thing.)The capture of each line of pixels in frame 102 is the same the captureof line zero. Similarly, each frame is captured in the same manner asframe 102. All the lines are not captured at the same time, e.g., theimage capture device does not have a mechanical shutter that stops lightfrom reaching a pixel after a predetermined time. Rather, each row ofpixels is sequentially read out. This is indicated by diagonal line102_S for frame 102. The round dot at the right end of each horizontalline indicates that the row select line goes active so that the value ofeach pixel in the row can be read out on the read line for that row.

To allow the pixel to again accumulate charge over a known timeinterval, the signal on the reset line for each pixel in row zero goesactive and sets each pixel to a known state.

Following the active reset signal, a pixel accumulates chargecorresponding to the light incident on the pixel until a signal on therow zero select line goes active, and then the charge stored in thepixel is available on the read line associated with the row. Each row inthe frame is read in the same way. When all the rows have been read, ablank row is read to allow for defining the frame. The blank row assuresthat the loads on the power supply remain constant, and so reduces noisein the captured frames.

SUMMARY

Video viewing capability of a device is enhanced by incorporating anenhanced frame timer in the device to increase the sensitivity of bothvisible scenes and alternate modality scenes. For example, astereoscopic image capture device includes a first image sensor, asecond image sensor, a first frame timer, and a second frame timer. Thefirst and second frame timers are different frame timers. The firstimage sensor includes a first plurality of rows of pixels. The secondimage sensor includes a second plurality of rows of pixels. The firstand second image sensors can be separate devices or different areas of asensor region in an integrated circuit. The first frame timer is coupledto the first image sensor to provide image capture timing signals to thefirst image sensor. The second frame timer is coupled to the secondimage sensor to provide image capture timing signals to the second imagesensor.

The dual frame timers provide many advantages. For example, one frametimer can be configured to provide signals to one of the image sensorsso that the image sensor captures frames at a normal video rate. Theother frame timer can be configured to provide signals to the other ofthe image sensors so that the other of the image sensors captures scenesat a rate slower than the normal video rate. This allows the other ofthe image sensors to integrate the available light over a longer periodof time, and so improve the signal to noise ratio. Specifically, in oneaspect, the first frame timer is configured to provide image capturetiming signals to sequentially capture N frames in the first imagesensor. The second frame timer is configured to provide image capturetiming signal to capture one frame in second image sensor for every Nframes captured in the first image sensor. Thus, each frame captured bythe second image sensor integrates the incident light for a longerperiod of time than does the first image sensor. This can also beaccomplished if the first frame timer is configured to expose each rowof the first plurality of rows of pixels for a first exposure time andif the second frame timer is configured to expose each row of the secondplurality of active of pixels for a second exposure time, where thefirst exposure time is different from the second exposure time.

An improved signal to noise ratio also can be obtained with multiplepixel binning. In this aspect, the first image sensor of thestereoscopic image capture device includes, for example, a Bayer colorfilter array over the first plurality of pixel rows. Each location ofthe first plurality of pixel rows of the first image sensor includes aset of Bayer pixels. The first framer timer circuit is configured tocombine each set of Bayer pixels in a row to form a single output pixel.

In one aspect, the multiple pixel binning is used in combination withthe longer exposure time for one of the image capture sensors, sometimescalled image sensors. For example, the first image sensor of thestereoscopic image capture device includes a Bayer color filter arrayover the first plurality of pixel rows. Each location of the firstplurality of pixel rows of the first image sensor includes a set ofBayer pixels. The first framer timer circuit is configured to combineeach set of Bayer pixels in a row to form a single output pixel. Thefirst frame timer also is configured to expose each row of the firstplurality of active rows of pixels for a first exposure time. The secondframe timer is configured to expose each row of the second plurality ofpixel rows for a second exposure time. The first exposure time isdifferent from the second exposure time. This is advantageous, forexample, when it desired to superimposed an augmented scene, such as afluorescence scene, on a monochromatic scene of the surgical site.

In one aspect, the first plurality of rows of pixels includes aplurality of pixel cells. Each of the plurality of pixel cells includesa plurality of pixels. In this aspect, the first image sensor alsoincludes a visible light color filter array including a plurality ofdifferent individual visible light color filters and an alternativelight filter array including a plurality of individual alternative lightfilters. One individual alternative light filter of the plurality ofindividual alternative light filters covers both a first set of pixelsof a plurality of pixels in a first pixel cell of the plurality of pixelcells and a second set of pixels of a plurality of pixels in a secondpixel cell of the plurality of pixel cells. The first pixel cell isadjacent the second pixel cell. Each of the plurality of the differentindividual visible light color filters covers a different pixel in thefirst and second sets of pixels. The pixels covered by individualvisible light color filters of the plurality of different individualcolor filters are different from pixels covered by the individualalternative light filter.

In this aspect, the first frame timer is configured to simultaneouslyreset pixels in the first and second pixel cells covered by one of thedifferent individual visible light color filter. The frame timer also isconfigured to simultaneously read a first pixel of the first pixel cellcovered by one of the plurality of different individual visible lightcolor filters and a second pixel of the second pixel cells covered byone of the plurality of different individual visible light colorfilters.

The first frame timer is also configured to simultaneously read a firstpixel in a first set of pixels of the plurality of pixels in a firstpixel cell of the plurality of pixel cells and a second pixel of asecond set of pixels of a plurality of pixels in a second pixel cell ofthe plurality of pixel cells. In this aspect, the image capture deviceis configured to bin the first read pixel and the second read pixel.

In another aspect, the first image sensor further includes a pluralityof visible light color filtered cells interleaved with a plurality ofalternative light filtered pixel cells.

In still another aspect, an image capture device includes an imagesensor. The image sensor includes a plurality of rows of pixels and avisible light color filter array. The visible light color filter arrayincludes a plurality of different individual visible light colorfilters. The plurality of rows of pixels includes a plurality of pixelcells. Each of the plurality of pixel cells includes a plurality ofpixels. Each pixel of the plurality of pixels of a pixel cell is coveredby a different one of the plurality of different individual visiblelight color filters. A frame timer is coupled to the image sensor toprovide image capture timing signals to the image sensor. The frametimer is configured to combine a plurality of pixels of a pixel cell toform a single output pixel.

In a further aspect, an image capture device includes an image sensorhaving a plurality of rows of pixels, a visible light color filterarray, and an alternative light filter array. The plurality of rows ofpixels includes a plurality of pixel cells. Each of the plurality ofpixel cells includes a plurality of pixels. The visible light colorfilter array includes a plurality of different individual visible lightcolor filters. The alternative light filter array includes a pluralityof individual alternative light filters. One individual alternativelight filter of the plurality of individual alternative light filterscovers both a first set of pixels of the plurality of pixels in a firstpixel cell of the plurality of pixel cells and a second set of pixels ofthe plurality of pixels in a second pixel cell of the plurality of pixelcells. The first pixel cell is adjacent the second pixel cell. Each ofthe plurality of the different individual visible light color filterscovers a different pixel in the first and second sets of pixels. Thepixels covered by individual visible light color filters of theplurality of individual visible light color filters being different frompixels covered by the individual alternative light filter.

The image capture device also includes a frame timer coupled to theimage sensor to provide image capture timing signals to the imagesensor. For example, the frame timer is configured to simultaneouslyreset pixels in the first and second pixel cells covered by one of theplurality of different individual visible light color filters. The frametimer also is configured to simultaneously read a first pixel of thefirst pixel cell covered by one of the plurality of different individualvisible light color filters and a second pixel of the second pixel cellscovered by one of the plurality of different individual visible lightcolor filters.

A first method includes exposing each row of a first plurality of rowsof pixels of a first image sensor of a stereoscopic image capture devicefor a first exposure time using signals from a first frame timer. Themethod also includes exposing each row of a second plurality of rows ofpixels of a second image sensor of the stereoscopic image capture devicefor a second exposure time using signals from a second frame timer,where the first exposure time is different from the second exposuretime.

Another method includes outputting a single output pixel from a locationin an image sensor including a plurality of Bayer pixels. The outputtingis by a frame timer using signals to combine the plurality of Bayerpixels at the location to form the single output pixel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a timing diagram for capture of a scene using a rollingshutter.

FIG. 2 is a diagram of a computer-assisted surgical system that includesan enhanced frame timer that allows implementation of alternativerolling shutter image capture sequences.

FIG. 3 is a more detailed diagram of part of a computer assistedsurgical system that includes a stereoscopic image capture device witheach channel of the stereoscopic image capture device having its ownframe timer.

FIG. 4 is a timing diagram for one aspect of capture of scenes withdifferent exposure times using a rolling shutter in the stereoscopicimage capture device of FIG. 3 .

FIG. 5 is a more detailed timing diagram for one aspect of capture ofscenes with different exposure times using a rolling shutter in thestereoscopic image capture device of FIG. 3 .

FIG. 6 is a generalized diagram showing that in one channel of thestereoscopic image capture device of FIG. 3 N frames are captured whileonly a single frame is captured in the other channel of the stereoscopicimage capture device of FIG. 3 .

FIG. 7 is a more detailed diagram of part of a computer assistedsurgical system that includes an image capture device with a singleimage sensor and a single frame timer.

FIG. 8A is diagram of a frame timer and a portion of a pixel array of animage sensor with a Bayer color filter array that includes a set ofBayer pixels, sometime referred to as a plurality of Bayer pixels, ateach location in the pixel array.

FIG. 8B is diagram of a frame timer and a portion of a pixel array of animage sensor with visible light color filter array and an alternativelight filter array.

FIG. 8C is diagram of another example of a frame timer and a portion ofa pixel array of an image sensor with a visible light color filter arrayand an alternative light filter array.

FIG. 9A is a timing diagram for multiple pixel binning at a location inthe pixel array of FIG. 8A.

FIG. 9B is a timing diagram for the image capture device of FIG. 8B.

FIG. 9C is a timing diagram for an unbinned pixel read and resetsequence of rows zero and one of the image capture device of FIG. 8C.

FIG. 9D is a timing diagram for a four-way binned hyperspectral pixelread and reset sequence of rows zero and one of the image capture deviceof FIG. 8C.

FIG. 9E is a timing diagram for an unbinned pixel read and resetsequence of rows two and three of the image capture device of FIG. 8C.

FIG. 9F is a timing diagram for a four-way binned hyperspectral pixelread and reset sequence of rows two and three of the image capturedevice of FIG. 8C.

FIG. 10 illustrate some of the combinations that can be obtained usingthe stereoscopic image capture device of FIG. 3 with dual frame timerlogic circuits and various timing sequences.

FIG. 11 illustrates various combinations of the frame timer timingsequences that can be implemented using the image capture device of FIG.7 .

In the drawings, the first digit of an element's reference numberindicates the figure with that single digit figure number in which theelement first appeared. The first two digits of an element's referencenumber indicate the figure with a double digit figure number in whichthe element first appeared.

DETAILED DESCRIPTION

Aspects of this invention augment video capturing capability and videoviewing capability of surgical devices, e.g., computer-assisted surgicalsystems such as the da Vinci® Surgical System commercialized byIntuitive Surgical, Inc. of Sunnyvale, Calif., by incorporating anenhanced frame timer to increase the sensitivity of both visible scenesand alternate modality scenes that are used to identify tissue or otheraspects of clinical interest during surgery. (da Vinci® is a registeredtrademark of Intuitive Surgical, Inc. of Sunnyvale, Calif.) While acomputer-assisted surgical system is used herein as an example, aspectsof this invention can be used with any device or system that utilizesalternate imaging modalities.

Enhanced frame timer 222 (FIG. 2 ) utilizes new types of pixel controlsequences, which, in one aspect, are implemented in digital logic at lowoverhead. These control sequences are designed to enhance sensitivityfor alternate imaging modes (hyperspectral, fluorescence, high dynamicrange, etc.).

A typical Complementary Metal-Oxide-Semiconductor (CMOS) sensor frametimer in an image capture system used a set of state machines to controlsignals on Reset, Transfer, and Row Select lines in an image sensorpixel array. These state machines typically output a simple sequence ofpulses, allowing for shutter width adjustment and things like sceneflip. The typical frame time circuits are designed around the particularpixel cell used (e.g., a four-way shared pixel cell), but forconventional imaging uses, access to the low-level timing signals is notallowed. The typical frame timer circuits only allow the user to selectvalues from a limited set of parameters, such as setting a shutter timeand row time or frame rate, and altering the timing for specific HighDynamic Range (HDR) modes. The four-way shared pixel cell referencedhere, allows many alternate timing sequences, but the conventional frametimer, designed for typical consumer uses, treats these shared pixelcells as a non-shared array and simply scans the pixels by row and bycolumn. In one aspect, enhanced frame timer 222 of image capture system220 includes enhanced fixed logic that allows more sequences of pulsesto be generated on Reset, Transfer and Row Select lines in an imagesensor 221 than was possible with prior CMOS sensor frame timers.

In another aspect, enhanced frame timer 222 is implemented with a softframe timer, where a pulse sequence is downloaded to a memory, e.g., aRAM block, and enhanced frame timer 222 reads the pulse sequence togenerate signals on Reset, Transfer and Row Select lines in image sensor221. This has the advantage that new sequences can be added after thesilicon for image sensor 221 including frame timer 222 is released.

Hence, aspects of the invention provide a new flexibility in an enhancedframe timer 222 associated with an image sensor 221. This flexibilityallows separation of the exposure for advanced imaging modes(hyperspectral, fluorescence, etc.) from that used in visible-lightimaging on image sensor 221. This, in turn, allows different tradeoffsto be made, like a slower frame rate for the advanced imaging data, toimprove sensitivity.

Another way enhanced frame timer 222 improves advanced imagingperformance is through the on-chip binning of pixels covered by a singlefilter element. On-chip binning offers noise reduction compared withseparate sampling and binning in the digital domain. A typical imagesensor bins pixels either for a monochrome sensor or for a Bayerpattern. However, hyperspectral filters available to put on image sensor221 are larger than typical image sensor pixel sizes. Consequently, withenhanced frame timer 222, the pixel cell is chosen to allow binningtailored to the desired filter pixel size and not just to the pixel sizeof image sensor 221.

In a stereo image sensor, two active areas on image sensor 221 areusually read out synchronously, which minimizes artifacts in displayedthree-dimensional video. However, when a stereo image sensor is used forcombined white-light imaging and advanced imaging, enhanced frame timer222 allows different exposures on the two active areas, while combiningthe captured pixel data onto a single stream for transmission.

Enhanced frame timer 222 also enables more sensitive advanced imaging bycovering different lines of an active area of image sensor 221, or oneof the active areas in a stereo image sensor, with a different filtermaterial; for example, one active area is set up for visible imaging,and the other active area with no filters, for fluorescence imaging.

Often, it is desirable to acquire advanced imaging data along withvisible-light scenes at the same time. Enhanced frame timer 222 utilizesa method for interlacing different exposure settings per color or perrow of the one image sensor with conventional video imaging. Thealternate pixel timing sequences used for the advanced imaging modes canalso be used for high dynamic range visible light imaging; for example,by exposing green pixels in a typical Bayer pattern differently.

Finally, the same frame timer enhancements used to enable advancedimaging can be applied to the standard Bayer pattern, by using afour-way shared pixel cell to allow a simple means of exposing differentcolors by different amounts, for improved noise performance inconventional imaging.

FIG. 2 is a high level diagrammatic view of a computer-assisted surgicalsystem 200, for example, the da Vinci® Surgical System. In this example,a surgeon, using a surgeon's console 214, remotely manipulates anendoscope 201 using a robotic manipulator arm 213. The surgeon can alsomanipulate surgical instruments mounted on other robotic manipulatorarms. There are other parts, cables etc. associated withcomputer-assisted surgical system 200, but these are not illustrated inFIG. 2 to avoid detracting from the disclosure. Further informationregarding computer-assisted surgical systems may be found, for example,in U.S. Patent Application Publication No. US 2008-0065105 A1 (filedJun. 13, 2007; disclosing Minimally Invasive Surgical System) and U.S.Pat. No. 6,331,181 (filed Dec. 18, 2001; disclosing Surgical RoboticTools, Data Architecture, and Use), both of which are incorporatedherein by reference.

An illumination system (not shown) is coupled to or alternativelyincluded within endoscope 201. In one aspect, the illumination systemprovides white light illumination or a combination of white lightillumination and an alternate imaging mode illumination, e.g.,hyperspectral illumination. In one aspect, all or part of this light iscoupled to at least one illumination path in endoscope 201. In anotheraspect, the illumination sources are located at or near the distal tipof endoscope 201. In one aspect, both the visible white lightillumination and the alternate imaging mode illumination are constantduring the surgical procedure. In another aspect, the visibleillumination is constant in time, but the spectrum of the alternateimaging mode illumination changes with time.

In this aspect, light from endoscope 201 illuminates tissue 203 of apatient 211. Endoscope 201, in one aspect, is a stereoscopic endoscope,which includes two optical channels, e.g., a left optical channel and aright optical channel, for passing light from tissue 203 to image sensor221, which includes two sensing areas—one that captures a left scene andanother that captures a right scene. Endoscope 201, in another aspect,is a monoscopic endoscope, which includes a single optical channel forpassing light from the tissue to image sensor 221, which in thisinstance includes a single sensing area.

As explained more completely below, for both types of endoscopes, thereflected white light is captured as a visible light frames by an imagecapture system 220. Visible light frames include, for example, visiblescenes that include scenes of tissue, and visible light frames sometimesare referred to as visible frames. Reflected non-visible light and/oremitted light from tissue are captured as augmented light frames byimage capture system 220. Augmented light frames include, for examplehyperspectral scenes of tissue 203 or other features in the field ofview of endoscope 201, or fluorescence from tissue 203. In anotheraspect, the augmented light frames include pixels with differentexposures, which can be used in producing high dynamic range scenes.Augmented light frames sometimes are referred to as augmented frames.

In one aspect, cameras in image capture system 220 are mounted on aproximal end of endoscope 201. In another aspect, the cameras aremounted in a distal end of endoscope 201. Here, a camera includes atleast a frame timer and an image sensor. Here, distal means closer tothe surgical site and proximal means further from the surgical site. Thecameras capture the visible and augmented frames through the same frontend optics, in one aspect. This is contrast to systems that utilizespecial front end optics to capture for example hyperspectral frames.

FIG. 3 is a more detailed illustration of the aspects of one example ofcomputer-assisted surgical system 200 of FIG. 2 . In the embodiment ofFIG. 3 , computer-assisted surgical system 200 includes an illuminatorthat is a combination light source 310. Combination light source 310includes a visible light illuminator 311, e.g., a white light source,and an augmented light illuminator 312. The particular implementation ofilluminators 311 and 312 is not critical so long as combination lightsource 310 has the capabilities described more completely below.

In this aspect, combination light source 310 is used in conjunction withat least one illumination path in stereoscopic endoscope 201 toilluminate tissue 203. In one aspect, combination light source 310 hasat least two modes of operation: a normal viewing mode and an augmentedviewing mode.

In the normal viewing mode, visible light illuminator 311 providesillumination that illuminates tissue 203 in white light. Augmented lightilluminator 312 is not used in the normal viewing mode.

In the augmented viewing mode, visible light illuminator 311 providesillumination that illuminates tissue 203 in white light. In one aspect,augmented light illuminator 312 provides illumination that illuminatestissue 203 with hyperspectral light, e.g., light in the near-infraredspectrum, or alternatively with light that excites fluorescence.

Use of near-infrared light as an example of hyperspectral illuminationis illustrative only and is not intended to be limiting to thisparticular aspect. In view of the disclosure, one knowledgeable in thefield can select hyperspectral illumination that makes the non-salientfeatures in the captured visible frames salient in the capturedaugmented frames.

In one aspect, visible light illuminator 311 includes a source for eachof the different visible color illumination components. For ared-green-blue implementation, in one example, the sources are lasers, ared laser, two green lasers and a blue laser. In one aspect, the lightfrom visible light illuminator 311 has its spectrum shaped so that thelight appears to have a purple tint to the human eye. See PCTInternational Publication No. WO 2015/142800 A1, which is incorporatedherein by reference.

The use of lasers in visible light illuminator 311 is illustrative onlyand is not intended to be limiting. Visible light illuminator 311 couldalso be implemented with multiple light emitting diode (LED) sourcesinstead of lasers for example. Alternatively, visible light illuminator311 could use a Xenon lamp with an elliptic back reflector and a bandpass filter coating to create broadband white illumination light forvisible scenes. The use of a Xenon lamp also is illustrative only and isnot intended to be limiting. For example, a high pressure mercury arclamp, other arc lamps, or other broadband light sources may be used.

The implementation of augmented light illuminator 312 depends on thelight spectrum of interest. Typically, a laser module, laser modules, alight-emitting diode or light emitting diodes are used as augmentedlight illuminator 312.

In the normal and the augmented viewing modes, the light from visiblelight illuminator 311 or light from visible light illuminator 311 andlight from augmented light illuminator 312 is directed into a connector316. Connector 316 provides the light to an illumination path instereoscopic endoscope 201 that in turn directs the light to tissue 203.Each of connector 316 and the illumination path in stereoscopicendoscope 201 can be implemented, for example, with a fiber opticbundle, a single stiff or flexible rod, or an optical fiber.

Light from surgical site 203 (FIG. 3 ) is passed by the stereoscopicoptical channel in endoscope 201, e.g., a left optical channel and aright optical channel, or alternatively, a first optical channel and asecond optical channel, to cameras 320L, 320R. The use of two discretecameras 320L and 320R is for ease of illustration and discussion, andshould not be interpreted as requiring two discrete cameras or twodiscrete image capture units. The components of cameras 320L and 320Rcan be combined in a single unit.

As explained more completely below, left camera 320L includes a leftimage sensor 321L. Left image sensor 321L captures light received fromthe left channel of stereoscopic endoscope 301 as a left frame 322L.Similarly, right camera 320R includes a right image sensor 321R. Rightimage sensor 321R captures light received from the right channel ofstereoscopic endoscope 301 as a right frame 322R. Left image sensor 321Land right image sensor 321R can be separate sensors or different activeareas of a single sensor. Also, the use of left and right is intended toassist in differentiating between the first and second sensors.

Camera 320L includes a first frame timer circuit 325L, sometimes calledframe timer 325L, which, in this aspect, is coupled to left cameracontrol unit 330L and to left image sensor 321L. Camera 320R includes asecond frame timer circuit 325R, sometimes called frame timer 325R,which, in this aspect, is coupled to right camera control unit 330R andto right image sensor 321R. The use of an individual frame timer foreach image sensor provides enhanced imaging capability compared toconfigurations that used a common frame timer for all the image sensors.The use of separate frame timers 325L, 325R allows separation of theexposure for advanced imaging modes (hyperspectral, fluorescence, etc.)captured by one of the image sensors from that used in visible-lightimaging on the other of the image sensors. This, in turn, allowsdifferent tradeoffs to be made, like a slower frame rate for theadvanced imaging data, to improve sensitivity. Another way the use ofseparate frame timers improves advanced imaging performance is throughon-chip binning of pixels covered by a single filter element. On-chipbinning offers noise reduction compared with separate sampling andbinning in the digital domain.

Camera 320L is coupled to a stereoscopic display 351 in surgeon'sconsole 214 by a left camera control unit 330L and image processingmodule 340. Image processing module 340 is a part of image processingsystem 130. Camera 320R is coupled to stereoscopic display 351 insurgeon's console 214 by a right camera control unit 330R and imageprocessing module 340. Camera control units 330L, 330R receive signalsfrom a system process control module 362. System process control module362 represents the various controllers in system 300.

Display mode select switch 352 provides a signal to a user interface 361that in turn passes the selected display mode to system process controlmodule 362. Various controllers within system process control module 362configure illumination controller 315, configure left and right cameracontrol units 330L and 330R to acquire the desired scenes, and configureany other elements in imaging processing module 340 needed to processthe acquired scenes so that the surgeon is presented the requestedscenes in stereoscopic display 351. Imaging processing module 340implements image processing pipelines equivalent to known imageprocessing pipelines.

The video output on stereoscopic display 351 may be toggled between thenormal and augmented viewing modes by using, e.g., a foot switch, adouble click of the master grips that are used to control the surgicalinstruments, voice control, and other like switching methods. The togglefor switching between the viewing modes is represented in FIG. 3 asdisplay mode select switch 352.

Central controller 360 and system process control module 362 are similarto prior systems with the exception of the aspects described morecompletely below. Although described as a central controller 360, it isto be appreciated that central controller 360 may be implemented inpractice by any number of modules and each module may include anycombination of components. Each module and each component may includehardware, software that is executed on a processor, and firmware, or anycombination of the three.

Also, the functions and acts of central controller 360 and systemprocess control module 362, as described herein, may be performed by onemodule, or divided up among different modules or even among differentcomponents of a module. When divided up among different modules orcomponents, the modules or components may be centralized in one locationor distributed across system 200 for distributed processing purposes.Thus, central controller 360 and system process control module 362should not be interpreted as requiring a single physical entity as insome aspects both are distributed across system 200.

Further information regarding computer-assisted surgical systems may befound for example in U.S. patent application Ser. No. 11/762,165 (filedJun. 23, 2007; disclosing Minimally Invasive Surgical System), U.S. Pat.No. 6,837,883 B2 (filed Oct. 5, 2001; disclosing Arm Cart forTelerobotic Surgical System), and U.S. Pat. No. 6,331,181 (filed Dec.28, 2001; disclosing Surgical Robotic Tools, Data Architecture, andUse), all of which are incorporated herein by reference.

In FIG. 3 , cameras 320L, 320R and combination light source 310 areshown as being external to endoscope 201. However, in one aspect,cameras 320L, 320R and light source 310 are included in the distal tipof endoscope 201, which is adjacent tissue 203. Also, left image sensor321L and right image sensor 321R can be different active areas of asensor region of an integrated circuit chip that includes left frametimer circuit 325L and right frame timer circuit 325R.

System controller 320 (FIG. 3 ) is illustrated as unified structures forease of illustration and understanding. This is illustrative only and isnot intended to be limiting. The various component of system controller320 can be located apart and still perform the functions described.

Stereoscopic Image Capture with Alternate Frame Timing

In some aspects, a first scene captured by left image sensor 321L ispresented in the left eye viewer of stereoscopic display 351 and asecond scene captured by right image sensor 321R is presented in theright eye viewer of stereoscopic display 351. For example, a normalcolor scene of the surgical site is presented to the left eye of theuser and an augmented scene of the surgical site is presented to theright eye of the user.

Typically, an augmented scene captured by one of the image sensors has asignificantly lower intensity than the intensity of a color scenecaptured by the other of the image sensors. Previously, the intensitydifferences were compensated for by digitally processing the capturedscenes. Unfortunately, this can introduce noise caused, for example, byamplifying the low signal levels.

In this aspect, frame timers 325L and 325R are configured to read outthe data from left image sensor 321L and the data from right imagesensor 321R at different rates. For example, as illustrated in FIG. 4 ,a visible color scene, i.e., a reflected white light scene, is capturedby left image sensor 321L at the normal rate, e.g., sixty frame persecond. An augmented scene, e.g., a fluorescence scene or ahyperspectral scene, is captured by right image sensor 321R at a slowerrate than the normal rate, e.g., thirty frame per second. FIG. 4illustrates the implementation of the rolling shutter by frame timer325L for left image sensor 321L and the implementation of the rollingshutter by frame timer 325R for right image sensor 321R.

In this example, each of left image sensor 321L and right image sensor321R is assumed to have (m+2) rows of pixels—(m+1) active rows and adummy row. Thus, the active rows are numbered from 0 to m.

Frame timer 325L repetitively provides signals on the transmit, resetand select lines to image sensor 321L captures each of frames 401L,402L, 403L, 404L, and 405L in the same way with same timing. In thisexample, the capture of frame 402L and in particular, row zero of frame402L is considered. The capture of each row of pixels in frame 402L isthe same as row zero.

As previously pointed out, with a rolling shutter, all the active rowsof image sensor 321L are not captured at the same time, e.g., camera320L does not have a mechanical shutter that stops light from reaching apixel after a predetermined time. Rather, each row of pixels issequentially read out. This is indicated by diagonal line 402L-S forframe 402L. Line 402L-S represents the rolling shutter for capture offrame 402L by image sensor 321L. Each of frames 401L, 403L, 404L, and405L has an equivalent rolling shutter 401L-S, 403L-S, 404L-S, and405L-S, respectively.

To allow each pixel in a row to again accumulate charge, a signal on thereset line for the row goes active. The square at the left end of eachhorizontal line in FIG. 4 represents the signal on the reset line forthat row going active. Thus, square 402L-0-RST_represents the resetsignal for row zero in frame 402L going active so that each pixel in rowzero is set to a known state and begins accumulating charge thatcorresponds to the light incident on that pixel.

The round dot at the right end of each horizontal line in FIG. 4indicates that the signal on the row select line for that row goesactive so that the value of each pixel in the row is read out. When eachpixel in the row is read out, the shutter for that row is effectivelyclosed. Thus, dot 402L-0-SLCT represents the row select signal for rowzero in frame 402L going active so that the value of each pixel in rowzero is read out.

Time 402L-0-EXP between when the time when pixels in row zero in frame402L are set to a known state and the time when the row select line forrow zero goes active and the pixel values in row zero are read out isthe exposure time for that row. Thus, frame timer 325L can control theexposure time for a row in a frame by controlling the time intervalbetween when the row select signal for the row in the previous framegoes active and when the reset signal for the row in the current framegoes active.

When all the active rows in frame have been read, a dummy row of imagesensor 321L is read. The time interval used in reading the dummy row istime interval 401L-BLNK for frame 401L, time interval 402L-BLNK forframe 402L, time interval 403L-BLNK for frame 403L, time interval404L-BLNK for frame 404L, and time interval 405L-BLNK for frame 405L.Blanking is a typical feature of video timing, but while blanking isuseful in processing and display, it is not necessary to have anyblanking or dummy row readout to use any of the pixel timing sequencesdescribed herein.

The operation of frame timer 325R with respect to resetting a row ofpixels and reading out a row of pixels is equivalent to that justdescribed for frame timer 325L, but the various signals are activated aslower rate. Frame 401R is captured in the same time interval thatframes 402L and 403L are captured, while frame 402R is captured in thesame time interval that frames 404L and 405L are captured.

Line 401R-S represents the rolling shutter for frame 401R. Square402R-0-RST represents the reset signal for row zero in frame 401R goingactive so that each pixel in row zero is set to a known state and beginsaccumulating charge that corresponds to the light incident on thatpixel. Dot 402R-0-SLCT represents the row select signal for row zero inframe 401R going active so that the value of each pixel in row zero isread out. Time 401R-0-EXP between when the time when pixels in row zeroin frame 401R are set to a known state and the time when the row selectline for row zero goes active and the pixel values in row zero are readout is the exposure time for that row.

When all the active rows in frame 401R have been read, a dummy row ofimage sensor 321L is read. The time interval used in reading the dummyrow is time interval 401R-BLNK for frame 401R.

Thus, FIG. 4 illustrates that while the frames in left image sensor 321Lare read out at the normal rate, the frames in right image sensor 321Rare read out at half the rate. This allows right image sensor 321R tointegrate in the incident light over a longer time period, which in turnimproves the signal to noise ratio compared to capturing the frames inright image sensor 321R at the normal rate and then digitally amplifyingthe captured signals.

FIG. 5 is a more detailed timing diagram of the reset and select signalsgenerated by frame timers 325L and 325R. Note that the timing diagram isfor the frames of interest in demonstrating the different exposure timesfor the two image sensors. FIG. 5 does not include all signals for theframes in FIG. 4 .

The references numerals of the pulses in FIG. 5 are the same as thecorresponding reference numerals in FIG. 4 . However, there are someadditional reference numerals in FIG. 5 . The key to the referencenumerals in FIGS. 4 and 5 is:

xxxy-s-name

-   where

xxx is the reference number of the frame in FIG. 4 ;

y represents that channel, Right or Left, in this example;

s is the row number, 0 to m for active rows, and D for the dummy row;and

name, RST=row reset, SLCT=row select, EXP=exposure time.

Frame timer 325L generates active row reset signals 401L-0-RST to401L-m-RST sequentially in time for each of rows zero to m of imagesensor 321L. Following the exposure time for each of the rows, frametimer 325L generate active row select signals 401L-0-SLCT to 401L-m-SLCTsequentially in time for each of rows zero to m of image sensor 321L.

After each of the active rows is reset, frame timer 325L generates anactive dummy row reset signal 401L-D-RST_for the dummy row of imagesensor 321L, and after the exposure time, frame timer 325L generates anactive row select signal 401L-D-SLCT for the dummy row of image sensor321L. After generating the dummy row signals, frame timer 325L continuesgenerating the row reset and row select signals for each of thesubsequent frames captured by image sensor 321L.

The operation of frame timer 325R is different from that of frame timer325L. Frame timer 325L generates active row reset signals 401R-0-RST to401L-m-RST_sequentially in time for each of rows zero to m of imagesensor 321R, but then frame timer 325R either stops generating activerow reset signals, or generates active reset signals for the dummy row,until it is time to initiate capture of the next frame.

After capture of a preceding frame in image sensor 321R is complete,frame timer 325R generates dummy row select signals 401R-D-SLCT untilexposure time 401R-0-EXP for the zeroth row in image sensor 321R haselapsed, and then frame timer 325R generates active row select signals401R-0-SLCT to 401L-m-SCLT sequentially in time for each of rows zero tom of image sensor 321R.

In this example, the exposure time for a frame captured by image sensor321R is twice as long as the exposure time for a frame captured by imagesensor 321L. However, this approach of using the two image sensors tocapture scenes with different exposures can be generalized asillustrated in FIG. 6 .

In FIG. 6 , frame timer 325L is configured to time sequentially captureN frames—frame 0 to frame (N−1)—in image sensor 321L while frame timer325R captures one frame—frame 0—in image sensor 321R. Here, N is apositive number greater than zero, in one aspect. Thus, the exposuretime of a frame captured in image sensor 321R is N times the exposuretime of a frame captured in image sensor. FIG. 5 is the case where N istwo.

Pixel Binning

The pixel binning aspects described more completely below can beimplemented in stereoscopic computer-assisted surgical system 200 ofFIG. 3 or in monoscopic system 700 of FIG. 7 . In FIG. 7 , image sensor321, image 322, and camera control unit 330 are equivalent to imagesensors 321R, 321L, frames 322R, 322L, and camera control units 330R,330L, and so the description of those elements is not repeated here.Similarly, image processing module 740, surgeon's console 714 withdisplay 751, central controller 760 and system process control module762 are equivalent to the corresponding element in FIG. 3 for either theleft or the right channels of FIG. 3 . Endoscope 701 is similar toendoscope 301, except endoscope 701 has only a single optical channelthat transmits light from tissue 203 to camera 720. Thus, monoscopicsystem 700 is equivalent to the system of FIG. 3 with one of the leftand right channels of FIG. 3 removed, and so is not described in furtherdetail because the description is repetitious of the description of theelements in FIG. 3 .

Multiple Pixel Binning with a Bayer Color Filter Array

FIG. 8A is an illustration of a representative portion of a Bayer colorfilter on a CMOS image sensor with a four-way shared pixel cell and anovel frame timer 825A. Thus, FIG. 8A is an example of a part of animage capture unit having an image sensor 821A with a Bayer color filterarray and a frame timer 825A. Image sensor 821A and frame timer 825A areexamples of image sensor 321L and frame timer 325L, image sensor 321Rand frame timer 325R, or image sensor 321 and frame trimer 325.

Each location in an image sensor includes a plurality of pixels. In FIG.8 , only four locations (0,0), (0,1), (1,0), and (1,1) are shown, whereeach location includes four pixels connected to a shared column line,which is a four-way shared pixel cell. The other locations in imagesensor 821A, which are not shown, are arranged in a correspondingmanner.

In this example, each pixel is covered by a filter in the Bayer colorfilter array. As is known, in a Bayer color filter array, fifty percentof the filters are green filters, twenty-five percent are red filters R,and twenty-five percent are blue filters B. In this example, the greenfilters are divided into first green filters Gr and second green filtersGb, for ease of discussion. The two green filters use the same filterdye and pass the same wavelength range, in this example. There is aone-to-one correspondence between filters in the Bayer color filterarray and the pixels in image sensor 821A, meaning that each pixel inimage sensor 821A is covered by a different filter in the Bayer colorfilter array, in this aspect. While a Bayer color filter array is usedas an example, it is not necessary that the color filter arrays havethis specific configuration. Color filter arrays with different colorsor with different proportions of the various colors can also be used inthe applications described herein.

A pixel covered by a red filter R is referred to as a red pixel R. Apixel covered by a first green filter Gr, is referred to a first greenpixel Gr. A pixel covered by a second green filter Gb, is referred to asecond green pixel Gb. A pixel covered by a blue filter B is referred toas a blue pixel B. Thus, in FIG. 8A, each location includes a red pixel,first and second green pixels, and a blue pixel. Also, in FIGS. 8A and8B, the rows are illustrated as extending in the vertical direction andthe columns as extending in the horizontal direction. This is for easeof illustration and should not be construed as limiting the rows andcolumns of the image sensor to any particular orientation. Theconfigurations described more completely below operate the sameindependent of the orientation of the rows and of the columns.

Each row driver of image sensor 821A is connected to a differentplurality of pixel rows. A first transmit line Tx_1 connects the rowdriver to each red pixel in a second row connected to the row driver. Asecond transmit line Tx-2 connects the row driver to each second greenpixel in a second row connected to the row driver. A third transmit lineTx_3 connects the row driver to each first green pixel in a first rowconnected to the row driver. A fourth transmit line Tx_4 connects therow driver to each blue pixel in the first row connected to the rowdriver.

A reset line RESET_connects the row driver to a shared column driverSHARED at each location in the two rows associated with the row driver.A select line SELECT_connects the row driver to shared column driverSHARED at each location in the two rows associated with the row driver.In one aspect, each shared column driver SHARED is a single floatingdiffusion charge storage node.

Frame timer 825A is connected to each of the row drivers of image sensor821A by a plurality of lines. In this example, the plurality of linesincludes twenty-one lines.

Ten lines of the twenty-one lines are row address lines ROW_ADDR<9,0>.Row address lines ROW_ADDR<9,0> carry an address of the row beingaccessed by frame timer 825A.

Three of the twenty-one lines are a row select line ROW_SELECT, a resetset line RST_SET, and a reset clear line RST_CLR. An active signal onrow select line ROW_SELECT_causes the row driver addressed by theaddress on row address lines ROW_ADDR<9,0> to drive an active signal onselect line SELECT. An active signal on reset set line RST_SET causesthe row driver addressed by the address on row address linesROW_ADDR<9,0> to drive an active signal on reset line RESET.

An active signal on reset clear line RST_CLR causes the row driveraddressed by the address on row address lines ROW_ADDR<9,0> to drive aninactive signal on reset line RESET.

Four of the twenty one lines are transmit set lines TX_SET<4,1> andanother four of the twenty-one lines are transmit clear linesTX_CLR<4,1>. Each one of transmit set lines TX_SET<4,1> is coupledthrough the row driver to a different one of first transmit line Tx_1,second transmit line Tx-2, third transmit line Tx_3, and fourth transmitline Tx_4—for example, transmit set line TX_SET(1) is coupled to firsttransmit line Tx_1, transmit set line TX_SET(2) is coupled to secondtransmit line Tx_2, etc. Similarly, each one of transmit set linesTX_CLR<4,1> is coupled through the row driver to a different one offirst transmit line Tx_1, second transmit line Tx-2, third transmit lineTx_3, and fourth transmit line Tx_4—for example, transmit clear lineTX_CLR(1) is coupled to first transmit line Tx_1, transmit clear lineTX_CLR(2) is coupled to second transmit line Tx_2, etc.

An active signal on transmit set line TX_SET(1) causes the row driveraddressed by the address on row address lines ROW_ADDR<9,0> to drive anactive signal on first transmit line Tx_1, and so on for the othertransmit set lines. An active signal on transmit clear line TX_CLR(1)causes the row driver addressed by the address on row address linesROW_ADDR<9,0> to drive an inactive signal on first transmit line Tx_1,and so on for the other transmit lines.

With reset set line RST_SET, a reset clear line RST_CLR, transmit setlines TX_SET<4,1>, and transmit clear lines TX_CLR<4,1>, pulses can besent to different rows during a single row time, and the length of thosepulses can be longer than the time between them. When transmit set lineTX_SET1 goes active, the particular first transmit line Tx_1 line withthe matching row address goes active and stays high until the same firsttransmit line Tx_1 is again addressed and transmit clear line TX_CLR1goes active. Lines TX_SETx and TX_CLRx, and RST_SET and RST_CLR aredriven with short pulses that control the timing of the edges of alonger pulse on line Tx_1, etc. Thus, with these lines, pulses can besent to different rows during a single row time, and the length of thosepulses can be longer than the time between them.

In this example, the timing uses a particular type of row drivercircuit, one that addresses each row and generates the pulses that go tothe pixel row control lines TXn, SEL and RESET using latches on each rowsignal. There are other ways this logic could be implemented;specifically, the same timings of the pixel control lines can begenerated by other types of logic and the same concepts would apply.

Also, in these examples a four-way shared pixel cell is used, where theoutput portion of the four-way shared pixel cell is shared among thefour pixels in a Bayer group. This is particularly useful for alternateframe timings, but the examples presented herein also could be appliedto other pixel sharing arrangements.

The layout of the pixel array, the row drivers, the input lines to therow drivers, and the output lines of the row drivers of image sensor821A are known, and so are not described in greater detail herein. Anovel aspect is the sequence of the signals provided on then input linesto image sensor 821A by frame timer 825A that provide enhanced imagesensor timing and as a consequence enhanced imaging capabilities.

FIG. 8A is representative of an image capture device that includes animage sensor coupled to a frame timer. The image sensor includes aplurality of rows of pixels and a visible light color filter array. Thevisible light color filter array includes a plurality of differentvisible light color filters, which are represented in FIG. 8A by a red,two greens, and a blue visible light color filters. The plurality ofrows of pixels includes a plurality of pixel cells with each of theplurality of pixel cells including a plurality of pixels. In the exampleof FIG. 8A, the pixel cells are identified by locations (0,0), (0,1),(1,0), and (1,1). Each pixel of the plurality of pixels of a pixel cellis covered by a different one of the plurality of different visiblelight color filters. In the example of FIG. 8A, each of the plurality ofpixels of the pixel cell at locations at location (0,0) is covered byone of the red, two greens, and a blue visible light color filters. Theframe timer is coupled to the image sensor to provide image capturetiming signals to the image sensor.

FIG. 9A illustrates a timing diagram for pixel binning of the fourpixels at a location in a row of FIG. 8A as part of a rolling shutter.In this aspect, frame timer 825A simultaneously transmits an activesignal on each of transmit set lines TX_SET<4,1> and an active signal onreset set line RST_SET. In response to these signals, the addressed rowdriver simultaneously drives an active transmit signal of each of firsttransmit line Tx_1, second transmit line Tx-2, third transmit line Tx_3,and fourth transmit line Tx_4 and an active reset signal on line RESET,as illustrated in FIG. 9A.

To read the pixels, after the appropriate exposure time, frame timer825A simultaneously transmits an active signal on each of transmit setlines TX_SET<4,1> and an active signal on row select line ROW_SELECT. Inresponse to these signals, the addressed row driver simultaneouslydrives an active transmit signal of each of first transmit line Tx_1,second transmit line Tx_2, third transmit line Tx_3, and fourth transmitline Tx_4 and an active signal on row select line SELECT, as illustratedin FIG. 9A.

Since all four of the pixels at a location are simultaneously connectedto the shared column line, e.g., read out simultaneously, thisintegrates the four pixels in the analog stage, which improves thesignal to noise level relative to doing the same integration during thedigital processing stage. Combining the pixels like this makes atradeoff; all color information is lost, as is some spatial resolution,in exchange for a reduction by 50% in the noise level.

If a single image sensor is being used as in FIG. 7 , this pixel binningcan be used to improve the signal to noise ratio of the captured scenes.If stereoscopic image sensors are being used, as in FIG. 3 , one imagesensor can be used to capture color scenes at the normal frame rate, andthe other sensor can be used to capture scenes at a slower frame ratealong with the pixel binning. The reset and select signals for thesensor with the lower frame rate, as shown for example in FIG. 5 , aregenerated for each row as described with respect to FIG. 9A so that thelower frame rate and pixel binning are combined. Alternatively, ifstereoscopic images sensors are being used, in one aspect, both imagesensors capture frames at the same frame rate, e.g., the normal framerate, but one of image sensors uses pixel binning. Hence, for each frametime interval, a full spatial resolution color frame is captured alongwith a monochromatic frame with a lower noise level. Both frames includethe same scene and the spatial relationship between the two frames isknown.

Multiple Pixel Binning with Visible Light Color Filter Array and anAlternative Light Filter Array

In other aspects, interleaved arrays of visible-light color andalternative light (hyperspectral or other wavelength band) filters onCMOS image sensors with a four-way shared pixel cell and a novel frametimer are used.

Herein, an alternative light filter refers to a filter that filtersother than visible light. The alternative light filter includes aplurality of individual alternative light filters, where eachalternative light filter is configured to cover one or more image sensorpixels, and typically is configured to cover a plurality of image sensorpixels. Sometimes, an individual alternative light filter is referred toas a pixel of the alternative light filter. Similarly, a visible lightcolor filter array includes a plurality of different individual visiblelight color filters.

Visible color filter arrays, e.g., Bayer color filter arrays, usingorganic dye are well-known, and can be applied to small (<2 μm) pixels.Other filter technologies, capable of selecting other wavelengths,narrow bands of wavelengths, or polarization of light are alsowell-known, but cannot currently be applied to small pixel structuresfound in typical image sensors because of the manufacturing processesrequired for those alternative filters cannot produce filter pixel sizescomparable to the pixel sizes of the image sensors. Typically, the pixelsize of an alternative filter is a multiple of the pixel size of theimage sensor.

To overcome this problem for an image sensor used in an endoscope, asingle image sensor is used to capture both conventional color imagesand images in other wavelength bands using a filter structure such asthat illustrated in FIG. 9B. To compensate for the larger pixel size ofthe alternative light filter, single pixel red-green-blue (RGB) filtersare interleaved with an array of individual alternative light filters,where, in this example, each individual alternative light filter coversa two-by-two pixel cell of the image sensor.

With the particular structure of an image sensor having four-way sharedpixel cells, a matching arrangement of the filter array, and the use ofspecific timing sequences in the sensor's frame timer, noise benefitsfor the alternative filter signals can be obtained, without sacrifice tothe noise or frame rate of the RGB pixels in the image sensor array.This operation makes use of the four-way shared pixel connections. Asnoted above with respect to FIG. 8A, the four-way shared pixel cellshares a single floating diffusion charge storage node among a group offour pixels.

The floating diffusion charge storage node SHARED, sometimes referred toas shared column driver SHARED, can be reset by pulsing reset lineRESET, and can be buffered and connected to the column output line bypulsing select line SELECT. Floating diffusion charge storage nodeSHARED can also be connected to any or all of the four surroundingpixels by pulsing the corresponding transmit line or lines.

Since the four-way shared pixel cell has the flexibility to connect anyof the four surrounding pixels to floating diffusion charge storage nodeSHARED (and hence, to the reset and/or the output), the connections canbe made in the pixels on the transmit lines using pulses from frametimer 825B such that when each transmit line connected to one row ofpixels is pulsed, the pixels in the row connected to floating diffusioncharge storage node SHARED are connected in the pattern:

TX_1: 1- - 1 1- - 1 1- - . . . - - 1

TX_2: - 2 2- - 2 2- - 2 2 . . . 2 2-

where TX_1 refers to a transmit line to one of the rows in the pluralityof four-way shared pixel cell and TX_2 refers to a transmit line to theother of the rows in the plurality of four-way shared pixel cell. Thus,as shown in FIG. 8B, the filters are then deposited in a staggeredpattern, so that the four colors in the Bayer array are split betweentwo different pixel-sharing cells. Because the two color pixels in eachrow are connected to different floating diffusion charge storage nodesSHARED, the two color pixels can be read at the same time using theappropriate timing sequence.

The splitting of the four colors in the Bayer array between twodifferent pixel-sharing cells forces the alternative-filter pixels alsoto be split, but the charge in the two pixels connected to a column canbe combined in floating diffusion charge storage node SHARED duringreadout without adding extra noise. The pairs of columns correspondingto a single filter location can be combined as voltages at the output ofthe column amplifiers (before the signal is digitized). The net resultis a lower-noise readout of the alternative-filter pixels, without lossof spatial or temporal resolution of the other pixels in the array.Hence, in this example, the individual hyperspectral filters making upthe hyperspectral filter array are staggered with respect to theindividual portions of the color array filter, so that two rows' worthof charge on the hyperspectral pixels can be binned when one row isread, but the color pixels can be read separately and are not binned.

In addition to the charge-domain selective pixel binning describedabove, it is also possible to extend the exposure time selectively, bysimilar pulse sequences of transmit lines TX_x, which omit certain resetand read sequences for transmit lines TX_x that go to pixels with theindividual alternative light filters.

Thus, FIG. 8B is an illustration of a representative portion of a Bayercolor filter array and an alternative light filter array, e.g., ahyperspectral filter array, on a CMOS image sensor with four-way sharedpixel cells and a novel frame timer 825B. Frame timer 825B, in thisexample, is configured to generate the pulse sequences illustrated inFIG. 9B.

FIG. 8B is an example of a portion of an image capture unit having animage sensor 821B with a Bayer color filter array and an alternativelight filter array and a frame timer 825B. Image sensor 821B and frametimer 825B also are examples of image sensor 321L and frame timer 325L,image sensor 321R and frame timer 325R, or image sensor 321 and frametrimer 325.

Each location in image sensor 821B includes a plurality of pixels. InFIG. 8B, only six locations (0,0), (0,1), (0,2), (1,0), (1,1), (1,2),are shown, where each location includes four pixels connected to ashared column line, which is a four-way shared pixel cell. The otherlocations in image sensor 821B, which are not shown, are arranged in acorresponding manner.

In this example, some pixels in a four-way shared pixel cell are coveredby a filter in the Bayer color filter array, while other pixels in thefour-way shared pixel cell are covered by a filter in an alternativelight filter array. As noted above, a pixel covered by a red filter R ofthe Bayer color filter array is referred to as a red pixel R. A pixelcovered by a first green filter Gr of the Bayer color filter array, isreferred to a first green pixel Gr. A pixel covered by a second greenfilter Gb of the Bayer color filter array, is referred to a second greenpixel Gb. A pixel covered by a blue filter B of the Bayer color filterarray is referred to as a blue pixel B.

Pixels in a group of pixels covered by an individual alternative lightfilter of the alternative light filter array are represented by a samereference numeral Pj, where j is an integer number, and are referred toas alternative light filtered pixels. As indicated above, in thisexample, each individual alternative light filter occupies a two-by-twopixel cell of image sensor 821B, but pixels Pj are split betweenadjacent four-way shared pixel cells. Thus, in image sensor 821B, eachfour-way shared pixel cell at locations (0,0), (0,1), (0,2), (1,0),(1,1), (1,2) includes a plurality of visible light color filtered pixelsand a plurality of alternative light filtered pixels.

In particular, four-way shared pixel cell at location (0,0) includes ared pixel R, a first green pixel Gr, and two alternative light filteredpixel P1, P1. Four-way shared pixel cell at location (0,1) includes ablue pixel B, a second green pixel Gb, and two alternative lightfiltered pixel P1, P1. Thus, as described above, the four Bayer filteredpixels, red pixel R, first green pixel Gr, second green pixel Gb andblue pixel B are split between the two adjacent four-way shared pixelcells. Similarly, two of the pixels P1, P1 covered by a singlealternative light filter array pixel are in each of the two adjacentfour-way shared pixel cells.

Each row driver of image sensor 821B is connected to a plurality ofpixel rows. A first color transmit line COLOR_Tx0 connects Row Driver 0to each blue pixel B and to each first green pixel Gr in a first row—row0—of image sensor 821B. A first alternative filter transmit line HYP_Tx0connects Row Driver 0 to each alternative light filtered pixel in thefirst row. A second color transmit line COLOR_Tx1 connects Row Driver 0to each red pixel R and to each second green pixel Gb in a secondrow—row 1—of image sensor 821B. A second alternative filter transmitline HYP_Tx1 connects Row Driver 0 to each alternative light filteredpixel in the second row.

A third color transmit line COLOR_Tx2 connects Row Driver 1 to each bluepixel B and to each first green pixel Gr in the third row—row 2—of imagesensor 821B. A third alternative filter transmit line HYP_Tx2 connectsRow Driver 1 to each alternative light filtered pixel in the third row.A fourth color transmit line COLOR_Tx3 connects Row Driver 1 to each redpixel R and to each second green pixel Gb in a fourth row—row 3—of imagesensor 821B. A fourth alternative filter transmit line HYP_Tx3 connectsRow Driver 1 to each alternative light filtered pixel in the fourth row.The line arrangement connecting Row Drivers 0 and 1 to the pixel rows isrepeated down the column of image sensor 821B.

Thus, the transmit lines are connected to pixels in adjacent pixel rowswith the patterns by frame time 825B providing appropriate pulses, asdescribed above, i.e.:

COLOR_Tx0 1 - - 1 1 - - 1 1 - - . . . - - 1 HYP_Tx0 - 2 2 - - 2 2 - - 22 . . . 2 2 - COLOR_Tx1 1 - - 1 1 - - 1 1 - - . . . - - 1 HYP_Tx1 - 22 - - 2 2 - - 2 2 . . . 2 2 - COLOR_Tx2 - 2 2 - - 2 2 - - 2 2 . . . 22 - HYP_Tx2 1 - - 1 1 - - 1 1 - - . . . - - 1 COLOR_Tx3 - 2 2 - - 22 - - 2 2 . . . 2 2 - HYP_Tx3 1 - - 1 1 - - 1 1 - - . . . - - 1

A first reset line RESET_01 connects Row Driver 0 to a shared columndriver SHARED at each location in the first and second pixel rows. Afirst select line SELECT_01 connects Row Driver 0 to shared columndriver SHARED at each location in the first and second pixel rows. Asexplained above, in one aspect, each shared column driver SHARED is asingle floating diffusion charge storage node.

A second reset line RESET_connects Row Driver 1 to a shared columndriver SHARED at each location in the third and fourth pixel rows. Asecond select line SELECT_connects Row Driver 1 to shared column driverSHARED at each location in the third and fourth pixel rows.

Frame timer 825B is connected to each of the row drivers of image sensor821A by a plurality of lines. In this example, the plurality of linesincludes twenty-one lines.

Ten lines of the twenty-one lines are row address lines ROW_ADDR<9,0>.Row address lines ROW_ADDR<9,0> carry an address of the row beingaccessed by frame timer 825B.

Three of the twenty-one lines are a row select line ROW_SELECT, a resetset line RST_SET, and a reset clear line RST_CLR. An active signal onrow select line ROW_SELECT_causes the row driver addressed by theaddress on row address lines ROW_ADDR<9,0> to drive an active signal onthe select line. An active signal on reset set line RST_SET causes therow driver addressed by the address on row address lines ROW_ADDR<9,0>to drive an active signal on the reset line.

An active signal on reset clear line RST_CLR causes the row driveraddressed by the address on row address lines ROW_ADDR<9,0> to drive aninactive signal on the reset line.

Four of the twenty one lines are transmit set lines TX_SET<4,1> andanother four of the twenty-one lines are transmit clear linesTX_CLR<4,1>. Each one of transmit set lines TX_SET<4,1> is coupledthrough the row driver to a different one of the first transmit line,the second transmit line, the third transmit line, and the fourthtransmit line connected to the addressed row driver.

An active signal on transmit set line TX_SET(1) causes the row driveraddressed by the address on row address lines ROW_ADDR<9,0> to drive anactive signal on the first transmit line, and so on for the othertransmit set lines. An active signal on transmit clear line TX_CLR(1)causes the row driver addressed by the address on row address linesROW_ADDR<9,0> to drive an inactive signal on the first transmit line,and so on for the other transmit lines.

Thus, in image sensor 821B, the usual connections of the pairedtransmission lines to the pixels in each row, as illustrated in FIG. 8A,are permuted, so the pixels of like filter types (regular visible lightcolor filter array or alternative light filter array), are connected toseparate column drivers and readout circuits on each row. Because ofthis connection, separate timing control for the regular and alternativelight filter arrays are obtained. FIG. 9B is a timing diagram thatillustrates operations of image sensor 821B.

The example pulse sequence shown in FIG. 9B from frame timer 825Billustrates a reset of the pixels on pixel rows 0 and 1, followed laterby a readout of those pixels. Pixel rows 0 and 1 are the rows connectedto Row Driver 0. When the transmit pulse coincides with a reset pulse,both floating diffusion charge storage node SHARED and the photodiode(s)connected to floating diffusion charge storage node SHARED by activetransmit pulses are reset. When a reset pulse occurs alone, the resetpulse resets only floating diffusion charge storage node SHARED, whichis required for Correlated Double Sampling (CDS) to reduce readoutnoise.

The example pulse sequence in FIG. 9B:

1. Resets the color pixels on Row 0.

2. Resets the color pixels on Row 1.

3. Resets the alternative-filter pixels on Rows 0 and 1 together.

4. Later, reads the color pixels on Row 0.

5. Reads the color pixels on Row 1.

6. Reads the alternative-filter pixels on Rows 0 and 1, binned together.

Other exposures can be obtained by tuning the delay between reset andread sequences, and by selectively omitting reset/read sequences forsome pixel types.

Thus, FIGS. 8B and 9B are illustrative of an image capture deviceincluding an image sensor coupled to a frame timer. The image sensorincludes a plurality of rows of pixels, a visible light color filterarray, and an alternative light filter array. The plurality of rows ofpixels include a plurality of pixel cells. For example, the plurality ofpixel cells at locations (0,0), (0,1), (0,2), (1,0), (1,1), and (1,2) inFIG. 8B. Each of the plurality of pixel cells including a plurality ofpixels, which in the example of FIG. 8B is four pixels.

The visible light color filter array includes a plurality of differentvisible light color filters, which are represented in FIG. 8A by a red,two greens, and a blue visible light color filters. An alternative lightfilter array includes a plurality of individual alternative lightfilters. One individual alternative light filter of the plurality ofindividual alternative light filters covers both a first set of pixelsof a plurality of pixels in a first pixel cell of the plurality of pixelcells and a second set of pixels of a plurality of pixels in a secondpixel cell of the plurality of pixel cells. The first pixel cell isadjacent the second pixel cell. See the pixel cells at locations (0,0),(0,1) for an example of an individual alternative light filter. Each ofthe plurality of the different individual visible light color filterscovers a different pixel in the first and second sets of pixels. Thepixels covered by individual visible light color filters of theplurality of individual visible light color filters are different frompixels covered by the individual alternative light filter.

The frame timer is coupled to the image sensor to provide image capturetiming signals to the image sensor. For example, frame timer isconfigured to simultaneously reset pixels in the first and second pixelcells covered by one of the plurality of different individual visiblelight color filters.

As indicated above, the binning of data and the use of a combination ofa visible light color filter array and an alternative filter array alsocan be implemented in other ways using a shared pixel cell. For example,FIG. 8C is an illustration of a representative portion of a Bayer colorfilter array and an alternative light filter array, e.g., ahyperspectral filter array, on a CMOS image sensor with four-way sharedpixel cells and a novel frame timer 825C. As noted previously, a Bayercolor filter arrays is an example of a visible light color filter array,and the use of a Bayer color filter array is not intended to limit thevisible light color filter array to the particular combination of colorfilters described. Also, FIG. 8C is an example of a portion of an imagecapture unit having an image sensor 821C with a Bayer color filter arrayand an alternative light filter array and a frame timer 825C. Imagesensor 821C and frame timer 825C also are examples of image sensor 321Land frame timer 325L, image sensor 321R and frame timer 325R, or imagesensor 321 and frame trimer 325.

Each location in image sensor 821C includes a plurality of pixels. InFIG. 8C, only six locations (0,0), (0,1), (0,2), (1,0), (1,1), (1,2) areshown, where each location includes four pixels connected to a sharedcolumn line, which is a four-way shared pixel cell. The other locationsin image sensor 821C, which are not shown, are arranged in acorresponding manner.

In this example, in a pair of rows, alternating four-way shared pixelcells are covered by a portion of visible light color filter array, andalternating four-way shared pixel cells are covered by an individualalternative light filter of an alternative light filter array. As notedabove, when the visible light color filter array is a Bayer color filterarray, the pixels in a four-way shared pixel cell covered by a portionof the Bayer color filter array. Specifically, a pixel covered by a redfilter R of the Bayer color filter array is referred to as a red pixelR. A pixel covered by a first green filter Gr of the Bayer color filterarray, is referred to a first green pixel Gr. A pixel covered by asecond green filter Gb of the Bayer color filter array, is referred to asecond green pixel Gb. A pixel covered by a blue filter B of the Bayercolor filter array is referred to as a blue pixel B. When all the pixelsin a four-way shared pixel cell are covered by a portion of a visiblelight color filter array, the pixels are referred to as a visible lightcolor filtered pixel cell.

Pixels in a four-way shared pixel cell covered by a portion of anindividual alternative light filter cell of the alternative light filterarray are represented by a same reference numeral

Pj, where j is an integer number, and are referred to as an alternativelight filtered pixel cell. As indicated above, in this example, eachindividual alternative light filter covers all the pixels in a four-wayshared pixel cell of image sensor 821B. Thus, in this example locations(0,0), (1,1), (0,2) have visible light color filtered pixel cells, whilelocations, (0,1), (1,0) and (1,2) have alternative light filtered pixelcells.

Each row driver of image sensor 821C is connected to a plurality ofpixel rows. In the prior examples, each row driver had two transmitlines connected to a row of pixels. In this example, each row driver hasfour transmit lines connected to a row of pixels. Hence, in thisexample, Row Driver 0 and Row Driver 1 from the earlier examples arecombined into a single Row Driver 0/1, etc.

A first transmit line TXA_0 connects Row Driver 0/1 to each first greenpixel Gr in a first row—row 0—of image sensor 821C, e.g., to everyfourth pixel in the first row starting with the first pixel. A secondtransmit line TXB_0 connects Row Driver 0/1 to each blue pixel B in thefirst row of image sensor 821C, e.g., every fourth pixel in the firstrow starting with the second pixel. A third transmit line TXC_0 connectsRow Driver 0/1 to each first alternative light filtered pixel Px-1(where x equals 1 to 3 in FIG. 8C) of each alternative light filteredpixel cell in the first row of image sensor 821C, e.g., to every fourthpixel in the first row starting with the third pixel. A fourth transmitline TXD_0 connects Row Driver Oil to each second alternative lightfiltered pixel Px-2 of each alternative light filtered pixel cell in thefirst row of image sensor 821C, e.g., every fourth pixel in the firstrow starting with the fourth pixel.

A fifth transmit line TXA_1 connects Row Driver Oil to each red pixel Rin a second row—row 1—of image sensor 821C, e.g., every fourth pixel inthe second row starting with the first pixel. A sixth transmit lineTXB_1 connects Row Driver Oil to each second green pixel Gb in thesecond row of image sensor 821C, e.g., every fourth pixel in the secondrow starting with the second pixel. A seventh transmit line TXC_1connects Row Driver Oil to each third alternative light filtered pixelPx-3 (where x equals 1 to 3 in FIG. 8C) of each alternative lightfiltered pixel cell in the second row of image sensor 821C, e.g., toevery fourth pixel in the second row starting with the third pixel. Aneighth transmit line TXD_1 connects Row Driver Oil to each fourthalternative light filtered pixel Px-4 of each alternative light filteredpixel cell in the second row of image sensor 821C, e.g., to every fourthpixel in the second row starting with the fourth pixel.

A first reset line RESET_01 connects Row Driver Oil to a shared columndriver SHARED at each location in the first and second pixel rows. Afirst select line SELECT_01 connects Row Driver 0/1 to shared columndriver SHARED at each location in the first and second pixel rows. Asexplained above, in one aspect, each shared column driver SHARED is asingle floating diffusion charge storage node.

With respect to Row Driver 2/3, a first transmit line TXA_2 connects RowDriver 2/3 to each first alternative light filtered pixel Px-1 (where xequals 1 to 3 in FIG. 8C) in a third row—row 2—of image sensor 821C,e.g., to every fourth pixel in the third row staring with the firstpixel. A second transmit line TXB_2 connects Row Driver 2/3 to eachsecond alternative light filtered pixel Px-2 of each alternative lightfiltered pixel cell in the third row of image sensor 821C, e.g., toevery fourth pixel in the third row staring with the second pixel. Athird transmit line TXC_2 connects Row Driver 2/3 to each first greenpixel Gr of each visible light color filtered pixel cell in the thirdrow of image sensor 821C, e.g., to every fourth pixel in the third rowstaring with the third pixel. A fourth transmit line TXD_2 connects RowDriver 2/3 to each blue pixel B of each visible light color filteredpixel cell in the third row of image sensor 821C, e.g., to every fourthpixel in the third row staring with the fourth pixel.

Continuing with respect to Row Driver 2/3, a fifth transmit line TXA_3connects Row Driver 2/3 to each third alternative light filtered pixelPx-3 (where x equals 1 to 3 in FIG. 8C) in a fourth row—row 3—of imagesensor 821C, e.g., to every fourth pixel in the fourth row staring withthe first pixel. A sixth transmit line TXB_3 connects Row Driver 2/3 toeach fourth alternative light filtered pixel Px-4 of each alternativelight filtered pixel cell in the fourth row of image sensor 821C, e.g.,to every fourth pixel in the fourth row staring with the second pixel. Aseventh transmit line TXC_3 connects Row Driver 2/3 to each red pixel Rof each visible light color filtered pixel cell in the fourth row ofimage sensor 821C, e.g., to every fourth pixel in the fourth row staringwith the third pixel. An eighth transmit line TXD_3 connects Row Driver2/3 to each second green pixel Gb of each visible light color filteredpixel cell in the fourth row of image sensor 821C, e.g., to every fourthpixel in the fourth row staring with the fourth pixel.

A second reset line RESET_23 connects Row Driver 2/3 to a shared columndriver SHARED at each location in the third and fourth pixel rows. Asecond select line SELECT_23 connects Row Driver 2/3 to shared columndriver at each location in the third and fourth pixel rows. Theconfiguration of Row Drivers 0/1 and 2/3 is repeated down the column,and so additional row drivers are not illustrated in FIG. 8C.

Frame timer 825C is connected to each of the row drivers of image sensor821A by a plurality of lines. In this example, the plurality of linesincludes twenty-one lines.

Ten lines of the twenty-one lines are row address lines ROW_ADDR<9,0>.Row address lines ROW_ADDR<9,0> carry an address of the row beingaccessed by frame timer 825C.

Three of the twenty-one lines are a row select line ROW_SELECT, a resetset line RST_SET, and a reset clear line RST_CLR. An active signal onrow select line ROW_SELECT_causes the row driver addressed by theaddress on row address lines ROW_ADDR<9,0> to drive an active signal onthe select line. An active signal on reset set line RST_SET causes therow driver addressed by the address on row address lines ROW_ADDR<9,0>to drive an active signal on the reset line.

An active signal on reset clear line RST_CLR causes the row driveraddressed by the address on row address lines ROW_ADDR<9,0> to drive aninactive signal on the reset line.

Four of the twenty one lines are transmit set lines TX_SET<4,1> andanother four of the twenty-one lines are transmit clear linesTX_CLR<4,1>. Each one of transmit set lines TX_SET<4,1> is coupledthrough the row driver to a different one of the first transmit line,the second transmit line, the third transmit line, and the fourthtransmit line connected to the addressed row driver.

An active signal on transmit set line TX_SET(1) causes the row driveraddressed by the address on row address lines ROW_ADDR<9,0> to drive anactive signal on the first transmit line, and so on for the othertransmit set lines. An active signal on transmit clear line TX_CLR(1)causes the row driver addressed by the address on row address linesROW_ADDR<9,0> to drive an inactive signal on the first transmit line,and so on for the other transmit lines.

In image sensor 821C, four transmission gate phases are needed to binthe alternative light filtered pixels two by two in the charge domain(which provides four time the signal without added noise) and to sampleall visible light color filtered pixels at full resolution (unbinned).As shown in FIG. 8C, to accomplish this, it is necessary to runduplicate row lines for each transmission gate through the pixel array.To assist in differentiating between the duplicate row lines, the linesare labeled as TXA_<row #>, and TXB_<row #> in FIG. 8C. Phase A(indicated by labels TXA_<row #>), as described above, goes to goes toevery fourth pixel in a row starting with the first pixel, while phase B(indicated by labels TXB_<row #>) goes to every fourth pixel in the rowstarting with the second one, and so on. In the four-way shared pixelcell of FIG. 8A, there are only two phases, and the row lines for eachphase connect to alternate pixels.

When the alternative light filtered pixel cells are interleaved amongthe visible light color filtered cells diagonally, as illustrated inFIG. 8C, instead of being arranged as a two column set of color pixels,a two column set of hyperspectral pixels, a two column set of colorpixels, etc., as illustrated in FIG. 8B, the pulse sequences for thebinned case are different on the odd row pairs than on the even rowpairs. Thus, in FIGS. 9C to 9F different timing diagrams are presented,one for the unbinned case (FIGS. 9C and 9E) and one for the case wherethe color information is full-resolution but the hyperspectral is binnedtwo by two (FIGS. 9D and 9F). FIGS. 9C and 9D show the timing sequencesfor the even row pairs (0/1, 4/5, 8/9, . . . ) and FIGS. 9E and 9F showthe timing sequences for the odd row pairs, (2/3, 6/7, 10/11, . . . ).The pulse timings for the unbinned case (FIGS. 9C and 9E) are the samefor the even row pairs and the odd row pairs, but for the binned case(FIGS. 9D and 9F), the pulse timings are different for the even rowpairs and the odd row pairs.

In the binned case, all four alternative light filtered pixels in afour-way shared cell are connected simultaneously to the shared columnline, and the color pixels are read out in full resolution based on thetiming diagrams of FIGS. 9D and 9F. In the unbinned cases, each pixel isread individually.

FIGS. 10 and 11 illustrate some of the combinations that can be obtainedusing the stereoscopic image capture device of FIG. 3 with dual frametimer logic circuits and the various timing sequences described above.As noted above, the stereoscopic image capture device includes two imagesensors that each capture a frame and each frame includes a scene.

First, a normal stereoscopic scene 1001 is obtained when each of theframe timers implements a rolling shutter with the same exposure timeper image sensor row. Alternatively, the left and right scenes 1002 canhave different exposure times. In this aspect, as illustrated in FIG. 6, one frame timer implements a rolling shutter with a first exposuretime, and the other frame timer implements a rolling shutter with asecond different exposure time.

With multiple pixel binning, one of the two scenes produced is amonochromatic scene 1003. One frame timer for an image sensor with aBayer color filter array uses a rolling shutter and outputs a singlepixel for each location in a row of the image sensor. Each location is arow includes a plurality of Bayer pixels. See, for example, FIGS. 8A and9A.

Different exposure times and multiple pixel binning can be combined toproduce scenes with different exposure times and one of the scenes is amonochromatic scene 1104.

In the examples of FIG. 10 , a stereoscopic image capture device wasused. However, as illustrated in FIG. 11 , various combinations of theframe timer timing sequences described above can also be implementedusing the image capture device of FIG. 7 , which has a single frametimer logic circuit and a single image sensor. First, a normal scene1101 is obtained when the frame timer implements a rolling shutter withthe same exposure time per image sensor row.

With multiple pixel binning, the scene produced is a monochromatic scene1102. The frame timer for an image sensor with a Bayer color filterarray uses a rolling shutter and outputs a single pixel for eachlocation in a row of the image sensor. Each location is a row includes aplurality of Bayer pixels. See, for example, FIGS. 8A and 9A.

Herein, a computer program product includes a medium configured to storecomputer readable code needed for any one or any combination of methodsdescribed herein or in which computer readable code for any one or anycombination of the methods is stored. Some examples of computer programproducts are CD-ROM discs, DVD discs, flash memory, ROM cards, floppydiscs, magnetic tapes, computer hard drives, servers on a network andsignals transmitted over a network representing computer readableprogram code. A tangible non-transitory computer program productcomprises a medium configured to store computer readable instructionsfor any one of, or any combination of the methods described herein or inwhich computer readable instructions for any one of, or any combinationof the methods is stored. Tangible non-transitory computer programproducts are CD-ROM discs, DVD discs, flash memory, ROM cards, floppydiscs, magnetic tapes, computer hard drives and other physical storagemediums.

In view of this disclosure, instructions used in any one of, or anycombination of methods described herein can be implemented in a widevariety of computer system configurations using an operating system andcomputer programming language of interest to the user.

As used herein, “first,” “second,” “third,” etc. are adjectives used todistinguish between different components or elements. Thus, “first,”“second,” and “third” are not intended to imply any ordering of thecomponents or elements or to imply any total number of components orelements.

The above description and the accompanying drawings that illustrateaspects and embodiments of the present inventions should not be taken aslimiting—the claims define the protected inventions. Various mechanical,compositional, structural, electrical, and operational changes may bemade without departing from the spirit and scope of this description andthe claims. In some instances, well-known circuits, structures, andtechniques have not been shown or described in detail to avoid obscuringthe invention.

Further, this description's terminology is not intended to limit theinvention. For example, spatially relative terms—such as “beneath”,“below”, “lower”, “above”, “upper”, “proximal”, “distal”, and thelike—may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positions(i.e., locations) and orientations (i.e., rotational placements) of thedevice in use or operation in addition to the position and orientationshown in the figures. For example, if the device in the figures wereturned over, elements described as “below” or “beneath” other elementsor features would then be “above” or “over” the other elements orfeatures. Thus, the exemplary term “below” can encompass both positionsand orientations of above and below. The device may be otherwiseoriented (rotated 90 degrees or at other orientations) and the spatiallyrelative descriptors used herein interpreted accordingly. Likewise,descriptions of movement along and around various axes include variousspecial device positions and orientations.

The singular forms “a”, “an”, and “the” are intended to include theplural forms as well, unless the context indicates otherwise. The terms“comprises”, “comprising”, “includes”, and the like specify the presenceof stated features, steps, operations, elements, and/or components butdo not preclude the presence or addition of one or more other features,steps, operations, elements, components, and/or groups. Componentsdescribed as coupled may be electrically or mechanically directlycoupled, or they may be indirectly coupled via one or more intermediatecomponents.

All examples and illustrative references are non-limiting and should notbe used to limit the claims to specific implementations and embodimentsdescribed herein and their equivalents. Any headings are solely forformatting and should not be used to limit the subject matter in anyway, because text under one heading may cross reference or apply to textunder one or more headings. Finally, in view of this disclosure,particular features described in relation to one aspect or embodimentmay be applied to other disclosed

We claim:
 1. An image capture device comprising: a first image sensorcomprising a first plurality of rows of pixels, the first plurality ofrows of pixels comprising a plurality of pixel cells, each of theplurality of pixel cells comprising a plurality of pixels; a secondimage sensor comprising a second plurality of rows of pixels; a firstframe timer coupled to the first image sensor to provide image capturetiming signals to the first image sensor to cause the first image sensorto capture a first plurality of frames of pixel data at a first framerate; a second frame timer coupled to the second image sensor to provideimage capture timing signals to the second image sensor to cause thesecond image sensor to capture a second plurality of frames of pixeldata at a second frame rate different than the first frame rate; avisible light color filter array comprising a plurality of differentindividual visible light color filters; and an alternative light filterarray comprising a plurality of individual alternative light filters,one individual alternative light filter of the plurality of individualalternative light filters covering both a first set of pixels of aplurality of pixels in a first pixel cell of the plurality of pixelcells and a second set of pixels of a plurality of pixels in a secondpixel cell of the plurality of pixel cells, the first pixel cell beingadjacent the second pixel cell; each of the plurality of the differentindividual visible light color filters covering a different pixel in thefirst and second sets of pixels, the pixels covered by individualvisible light color filters of the plurality of different individualcolor filters being different from pixels covered by the individualalternative light filters.
 2. The image capture device of claim 1: theimage capture timing signals provided by the first frame timer areconfigured to cause the first image sensor to capture N frames of pixeldata, where N is greater than one; and the image capture timing signalsprovided by the second frame timer are configured to cause the secondimage sensor to capture only a single frame of pixel data while thefirst image sensor captures the N frames.
 3. The image capture device ofclaim 2: the first frame timer being configured to expose each row ofthe first plurality of rows of pixels for a first exposure time; thesecond frame timer being configured to expose each row of the secondplurality of rows of pixels for a second exposure time; and the firstexposure time being different from the second exposure time.
 4. Theimage capture device of claim 3: the first image sensor comprising aBayer color filter array, wherein each location of the first pluralityof rows of pixels of the first image sensor includes a set of Bayerpixels; and the first frame timer being configured to combine each setof Bayer pixels in a row to form a single output pixel.
 5. The imagecapture device of claim 4: the first frame timer being configured toexpose each row of the first plurality of rows of pixels for a firstexposure time; the second frame timer being configured to expose eachrow of the second plurality of rows of pixels for a second exposuretime; and the first exposure time being different from the secondexposure time.
 6. The image capture device of claim 1: the first frametimer being configured to simultaneously reset pixels in the first andsecond pixel cells covered by one of the different individual visiblelight color filters.
 7. The image capture device of claim 1: the firstframe timer being configured to simultaneously read a first pixel of thefirst pixel cell covered by one of the plurality of different individualvisible light color filters and a second pixel of the second pixel cellcovered by one of the plurality of different individual visible lightcolor filters.
 8. The image capture device of claim 1: the first frametimer being configured to simultaneously read a first pixel in a firstset of pixels of the plurality of pixels in a first pixel cell of theplurality of pixel cells and a second pixel of a second set of pixels ofa plurality of pixels in a second pixel cell of the plurality of pixelcells.
 9. The image capture device of claim 8, wherein the image capturedevice is configured to bin the first read pixel and the second readpixel.
 10. The image capture device of claim 1: the first image sensorfurther comprising: a plurality of visible light color filtered cellsinterleaved with a plurality of alternative light filtered pixel cells.11. A method comprising: providing a first image capture timing signalto a first image sensor of an image capture device to cause the firstimage sensor to capture a first plurality of frames of pixel data at afirst frame rate; and providing a second image capture timing signal toa second image sensor of an image capture device to cause the secondimage sensor to capture a second plurality of frames of pixel data at asecond frame rate different than the first frame rate, wherein: thefirst image sensor comprises a first plurality of rows of pixels; thefirst plurality of rows of pixels comprising a plurality of pixel cells,each of the plurality of pixel cells comprising a plurality of pixels;and the first image sensor further comprising: a visible light colorfilter array comprising a plurality of different individual visiblelight color filters; and an alternative light filter array comprising aplurality of individual alternative light filters, one individualalternative light filter of the plurality of individual alternativelight filters covering both a first set of pixels of a plurality ofpixels in a first pixel cell of the plurality of pixel cells and asecond set of pixels of a plurality of pixels in a second pixel cell ofthe plurality of pixel cells, the first pixel cell being adjacent thesecond pixel cell; each of the plurality of the different individualvisible light color filters covering a different pixel in the first andsecond sets of pixels, the pixels covered by individual visible lightcolor filters of the plurality of different individual color filtersbeing different from pixels covered by the individual alternative lightfilter.
 12. The method of claim 11, wherein: the first image capturetiming signal is configured to cause the first image sensor to capture Nframes of pixel data, where N is greater than one; and the second imagecapture timing signal is configured to cause the second image sensor tocapture only a single frame of pixel data while the first image sensorcaptures the N frames.
 13. The method of claim 11, wherein: the firstimage sensor comprises a first plurality of rows of pixels; the secondimage sensor comprises a second plurality of rows of pixels; the firstimage capture timing signal being configured to expose each row of thefirst plurality of rows of pixels for a first exposure time; the secondimage capture timing signal being configured to expose each row of thesecond plurality of rows of pixels for a second exposure time; and thefirst exposure time being different from the second exposure time. 14.The method of claim 11, wherein: the first image sensor comprises afirst plurality of rows of pixels; the second image sensor comprises asecond plurality of rows of pixels; the first image sensor comprising aBayer color filter array, wherein each location of the first pluralityof rows of pixels of the first image sensor includes a set of Bayerpixels; and the method further comprises combining each set of Bayerpixels in a row to form a single output pixel.
 15. The method of claim14, wherein: the first image capture timing signal being configured toexpose each row of the first plurality of rows of pixels for a firstexposure time; the second image capture timing signal being configuredto expose each row of the second plurality of rows of pixels for asecond exposure time; and the first exposure time being different fromthe second exposure time.
 16. The method of claim 11, further comprisingsimultaneously resetting pixels in the first and second pixel cellscovered by one of the different individual visible light color filters.17. A controller comprising: a processor; and a memory storingexecutable instructions that, when executed by processor, cause thecontroller to: provide a first image capture timing signal to a firstimage sensor of an image capture device to cause the first image sensorto capture a first plurality of frames of pixel data at a first framerate; and provide a second image capture timing signal to a second imagesensor of the image capture device to cause the second image sensor tocapture a second plurality of frames of pixel data at a second framerate different than the first frame rate, wherein: the first imagesensor comprises a first plurality of rows of pixels; the firstplurality of rows of pixels comprising a plurality of pixel cells, eachof the plurality of pixel cells comprising a plurality of pixels; andthe first image sensor further comprising: a visible light color filterarray comprising a plurality of different individual visible light colorfilters; and an alternative light filter array comprising a plurality ofindividual alternative light filters, one individual alternative lightfilter of the plurality of individual alternative light filters coveringboth a first set of pixels of a plurality of pixels in a first pixelcell of the plurality of pixel cells and a second set of pixels of aplurality of pixels in a second pixel cell of the plurality of pixelcells, the first pixel cell being adjacent the second pixel cell; eachof the plurality of the different individual visible light color filterscovering a different pixel in the first and second sets of pixels, thepixels covered by individual visible light color filters of theplurality of different individual color filters being different frompixels covered by the individual alternative light filter.
 18. Thecontroller of claim 17: the first image capture timing signal isconfigured to cause the first image sensor to capture N frames of pixeldata, where N is greater than one; and the second image capture timingsignal is configured to cause the second image sensor to capture only asingle frame of pixel data while the first image sensor captures the Nframes.
 19. The controller of claim 17, wherein the instructions, whenexecuted by the processor, cause the controller to: expose each row ofthe first plurality of rows of pixels for a first exposure time; andexpose each row of the second plurality of rows of pixels for a secondexposure time; wherein the first exposure time being different from thesecond exposure time.
 20. The controller of claim 17, wherein: the firstimage sensor comprising a Bayer color filter array, wherein eachlocation of the first plurality of rows of pixels of the first imagesensor includes a set of Bayer pixels; wherein the instructions, whenexecuted by the processor, cause the controller to combine each set ofBayer pixels in a row to form a single output pixel.